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(54) Method of providing background patterns for camera traddng 



(57) An improved method of tracking a camera (14) 
in dynamic chroma-keying of the type in which a fbre- 
ground object is photographed against a multicolored 
chroma-key screen. The multicolor patterns are created 
by projection (32,30) on a monochrome screen (20). for 
example by projecting light through a mask on a screen 
colored with the chroma-key color. The projection of 
multiple patterns can emulate the use of a single pattern 
having multiple levels of detail, and can enable lateral 
tracking of the camera. Sequentially projecting known 
transformations (for example, rotations) of a pattern 
adds an extra degree of freedom which further 
enhances the accuracy of the method. 
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Description 

FIELD AND BACKGROUND OF THE INVENTION 
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The present invention relates to video compositing s 
and. more particularly, to a method for providing a back- 
ground pattern so that the position and orientation of a 
moving camera can be determined in real time for the 
realistic compositing of images of a foreground object 
with a virtual t^ackground. io 

The technique of "chroma-key" compositing has 
long been used in video production to combine images 
of a foreground object with a virtual background stored 
in a digital data base. In this technique, the foreground 
object is photographed against a background of a is 
"chroma-key** color, typically a particular shade of blue. 
In the digital images thus captured, all the pixels colored 
with the chroma-key color are replaced with pixels 
derived from the virtual background. In order for the 
background pixels to be rendered in a way that provides 20 
a realistic composite image, the position and orientation 
of the camera must be known. 

Several method are known in the art for determin- 
ing the position and orientation of the camera. These 
methods can be classified in two categories. In methods 25 
of the first category, the position and orientation of the 
camera are tracked explicitly. In these methods, the 
camera must be provided with special tracking devices 
that are rigidly attached thereto. In some of the methods 
of the first category, the tracking devices are encoders. 30 
and lateral movement is tracked by encoders that are in 
physical contact with the floor of the studio. Another 
example of the methods of the first category, a magnetic 
tracker system, is taught t>y Loftus et al.. in PCT appli- 
cation no. US96/04846. Loftus et al.'s special tracking 35 
device is a magnetic tracker receiver rigidly attached to 
the camera. In the methods of the other category, the 
position and orientation of the camera are inferred from 
the captured images. For exanple, Graham, in US Pat- 
ent No. 5.502,482, which is incorporated by reference 40 
for all purposes as if fully set forth herein, teaches the 
use of a chroma-key background of two or more shades 
of tHue in a predetermined pattern, for ^cample a check- 
ertx>ard pattern. The position and orientation of the 
camera are inferred from the locations of features of the 45 
pattern on the captured images. 

Methods such as Graham's have the advantage of 
allowing the use of conventional video cameras, without 
special tracking equipment. This is particularly conven- 
ient for video compositing with hand-held cameras, to so 
avoid the necessity of contacting lateral motion encod- 
ers with the floor of the studio, or to avoid the extra bulk 
and weight of tracking equipment such as that of Loftus 
et al. Nevertheless, these methods have limitations of 
their own. One limitation is that the pattern is fixed, in ss 
fc)Oth level of detail and location. If the camera zooms in 
too closely on the foreground object, there may be too 
few features in the portion of the image occupying the 



camera's field of view for fWBcation and orientation of 
the camera to be determined accurately. Ck)nversely, if 
the camera is too far from the background pattern, the 
pattern features may be sufficiently crowded to make it 
difficult to distinguish between a panning motion of the 
camera and a lateral translation. In principle, the pattern 
can be provided with multiple levels of detail, to pre- 
serve tracking resolution at all required distances; but 
this adds to the complexity of the pattern. Similarly, if the 
foreground object is a live actor who moves laterally with 
respect to the pattern, and the camera follows the actor, 
the actor may move so far laterally that not enough of 
the pattern is left in the camera's f ieki of view to allow 
the camera's location and orientation to be determined 
accuracy. A second limitation is that the multiplication of 
chroma-key colors has been found to degrade the real- 
ism of the composite image. The nan'ower the spectral 
band of the pixels that are replaced with virtual back- 
ground pixels, the more realistic the resulting composite 
image. The loss of realism associated with the use of a 
multi-shade chroma-key background persists even 
while the camera is in a fixed position and only one 
chroma-key color would suffice. A third limitation is that 
the pattern is fixed in place in tiie studio, typically being 
painted on a wall or a fixed partition. This limits the flex- 
ibility of tiie metiiod and precludes its use, for example, 
in remote locations. 

There is thus a widely recognized need for, and it 
would be highly advantageous to have, a metiiod for 
providing an image for dynamic chroma-key composit- 
ing that is free of the limitations of the methods known in 
the art. 

SUMMAf^Y QF THE INVENTION 

According to the present invention there is provided 
an improved video production method of the type in 
which a plurality of images of an object in a foreground 
volume are captured successively by a camera and 
superposed on a virtual background, the position and 
orientation of the camera being determined using a first 
pattern featuring a first scale length and located at least 
partially beyond the object witii respect to the camera, 
the improvement including the steps of: (a) positioning a 
screen at least partially beyond the object with respect 
to the camera: and (b) projecting the first pattern onto 
the screen. 

Simply stated, according to the present invention, 
the pattern is projected on a screen behind the fore- 
ground object. By "screen" is meant herein any suitable 
flat or curved surface on which the pattern may be pro- 
jected, although this surface preferably is flat. As is 
noted below, the screen may be eltiier opaque or trans- 
parent. The screen and tiie projector may be easily con- 
figured to be portable, enabling the method to be used 
in remote locations. In one preferred embodiment, the 
two colors of the pattern are two shades of a base color, 
with the screen being colored with the base color and 
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the pattern being provided by Pt'^vBb white light on 
the screen through a mask, the shadow of the nnask 
thus creating an area on the screen characterized by a 
darker shade of the k>ase color than the part of the 
screen that is illuminated by the white light. If the cam- 5 
era is stationary for a prolonged period of time, or if a 
scene is shot in which a hand-hekl camera is not 
needed, so that a camera equipped with one of the spe- 
cial explicit tracking devices described above may be 
used, or ff a scene is shot in which the camera need not 10 
move at all, the projector is sinrply turned off. The color 
of the screen then is used as a single chroma-key color, 
preserving the enhanced realism of the composite 
image that is provided by the use of only one chroma- 
key color. IS 

Multiple projectors are used to project multiple pat- 
terns on the screen. In this way. a pattern having multi- 
ple levels of detail is emulated. For exanple, if the 
camera moves so close to the saeen that the level of 
detail of the first pattern is lost, a second pattern having 20 
a higher level of detail is superposed on the first pattern. 
Similariy, if the foreground object moves laterally, a sec- 
ond pattern is projected to the side of the first pattern, to 
provide continuous lateral coverage. 

25 

BRIEF DESCRIPTION OF THE DRAWINGS 

The invention is herein described, by way of exam- 
ple only, witii reference to the accompanying drawings, 
wherein: - 30 

FIG. 1 is a schematic partial depiction of a video 
studio configured according to the present inven- 
tion; 

FIG. 2 illustrates the use of multiple projected pat- 35 
terns to emulate a single pattern having multiple 
levels of dietail; 

FIG. 3 illustrates the use of multiple projected pat- 
terns to ti^ack a camera as the camera follows a lat- 
erally translating object. 40 

DESCRIPTION OF THE PREFERRED EMBODI- 
MENTS 

The present invention is of a metiiod of providing 45 
background patterns which can be used to determine 
the position and orientation of a mobile camera during 
chroma-key compositing. 

The principles and operation of camera tracking 
according to the present invention may be better under- so 
stood with reference to the drawings and tiie accompa- 
nying description. 

Referring now to the drawings. Figure 1 is a sche- 
matic partial illustration of a video studio configured 
according to the present invention. A camera 14 photo- ss 
graphs an object 1 0 against the background of a screen 
20. Object 1 0 is free to move within a foreground volume 
symbolized by dashed rectangle 12. Two projectors 30 
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and 32 are shown, ed^pBnf igured to project a pattern 
onto screen 20. Either but, not both projectors 30 and 
32 are used, each with a particular type of screen 20. If 
projector 30 is used, to project a pattern onto front sur- 
face 22 of screen 20 and obliquely relative to foreground 
volume 12. then screen 20 is opaque. If projector 32 is 
used, to project a pattern onto rear surface 24 of screen 
20. then screen 20 is transparent, so that the pattern 
projected onto rear surface 24 is visible on the other 
side of screen 20. Note that light 31 tiiat is projected on 
to front surface 22 must be projected obliquely relative 
to foreground volume 12 to avoid projecting light onto 
object 10. Thus, if screen 20 is vertical, as shown in Rg- 
ure 1 . back-projecting from projector 32 has the advan- 
tage that it is easier to obtain uniform illumination on 
screen 20 using light 33 that is projected substantially 
perpendicular to screen 20 than it is to obtain uniform 
illumination using ot^iquely projected light 31. 

The pattern projected onto saeen 20 is a geometric 
pattern chosen to facilitate the tracking of camera 14, 
i.e., the real time determination of the position and ori- 
entation of camera 14, from the images of the pattern 
behind object 10 as captured by camera 14. For exam- 
ple, the pattern may be a checkert>oard, as descrbed in 
US Patent No. 5,502.482 cited above. More generally, 
the pattern is any pattern tiiat enatrfes the position and 
oriientation of the camera to be inferred from the cap- 
tured images of the pattern. For example, tiie pattern 
may be a non-uniform checkerboard, i.e.. a grid of two 
rhutually perpendicular sets of parallel lines in which the 
lines of each set are spaced non-uniformly The compu- 
ter (not shown) that does the actual compositing is pro- 
vided with software to enable it to infer the position and 
orientation of camera 14, so that the background pixels 
in the images captured by camera 10 can be replaced 
realistically by virtual background pixels. For increased 
accuracy, at the start of a vkJeo session, the process is 
calibrated by providing the computer with initial values 
of the positional and orientational coordinates of cam- 
era 14. The first image captured by tiie camera then 
serves as a reference image, as described in the above- 
referenced US Patent No. 5.502,482. 

In general, each such pattern is characterized by a 
scale length. For example, in a non-uniform checker- 
board, the scale length is the average separation of two 
adjacent parallel lines. As noted above, the scale length 
of a pattern determines the range of distances between 
camera 14 and screen 20 over which tiiat pattern can 
be used to track camera 14 accurately Figure 2 is a 
frontal schematic illustration showing how multiple pro- 
jected patterns, characterized by two different scale 
lengtiis. are used to compensate for excursions of cam- 
era 14 outside tiiis range of distances. In Rgure 2. the 
patterns are represented by circles 40 and 42. This rep- 
resentation is symbolic: a real pattern would have a suit- 
ably conplicated geometry, such as the checkeit)oard 
described above, within the area of the circle. Initially, 
pattern 40 is projected onto screen 20 to enable the 
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tracking of camera 14. If camera 14 is sO^^o move so 
close to object 10 that pattem 40 has an insufficient 
level of detail to enable accurate tracking of camera 14. 
then a second pattern 42, having a scale length shorter 
than the scale length of pattern 40, is projected onto 5 
screen 20. Typically, the scale length of pattern 42 is 
half the scale length of pattern 40, allowing camera 14 
to approach within half the distance to screen 20 that 
would be allowed if only pattern 40 were used. Prefera- 
bly, patterns 40 and 42 are projected simultaneously 70 
onto screen 22 during the capture of at least one image 
by camera 14. In this way. the inferred position of cam- 
era 14. based on pattern 40 at the closest allowed dis- 
tance of approach of camera 14 to screen 20 based on 
pattern 40, is used to calibrate the first image including 15 
pattern 42 as a new reference image. 

Conversely, if camera 14 is about to withdraw so far 
from screen 20 that the scale length of pattern 40 is too 
short to allow accurate tracking of camera 14, then a 
pattern with a longer scale length, typically twice the 20 
scale length of pattern 40, is projected onto screen 20 to 
allow camera 14 to withdraw at least twice as far from 
screen 20 as would have been allowed using only pat: 
tern 40. This set of patterns, including patterns 40 and 
42 and other patterns whose scale lengths are the scale 25 
length of pattern 40 multiplied or divided by powers of 
two, thus serves as a nested set of self -similar patterns, 
enabling camera 14 to be positioned over a much wider 
range of distances from screen 20 than is possible 
under the prior art methods. 30 

Figure 3 is a frontal schematic illustration showing 
how multiple projected pattems are used to enable 
camera 14 to follow object 10 as object 10 moves later- 
ally with respect to screen 20. As in Figure 2. the pat- 
terns of Figure 3 are represented by circles 40 and 44. as 
Initially, object 1 0 is in front of pattern 40, as seen from 
camera 14. If object 10 moves to the left, to the position 
of object 1 0'. then not enough of pattern 40 is in the field 
of view of camera 1 4 to enable camera 1 4 to be tracked 
accurately. Before object 10 reaches the position of 40 
object 1 0'. a second pattern 44 is projected onto screen 
20. Pattern 44 is displaced far enough laterally from pat- 
tern 40 to allow the accurate tracking of camera 14 
when camera 14 is pointed at object 1 0*. As before, both 
patterns 40 and 44 are projected simultaneously onto 45 
screen 22 during the capture of at least one image by 
camera 14, to allow the first image of pattern 44 to be 
used as a reference image. 

As noted above, if screen 22 is opaque, then prefer- 
ably two different shades of the same t^se color are so 
used as chroma-key colors, and front surface 22 is 
colored with the base color. The pattern (for example, 
pattern 40) is created on surface 22 by projecting the 
image of a mask onto front surface 22: the shadowed 
areas then are the portion of the pattern that is colored ss 
with a darker shade of the base color, arKi the rest of the 
pattern retains the lighter shade of the base color. The 
light used to project the image of the mask may be 
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white, or may be the color ^^Pht surface 22. 

If the projected pattern is altered dynamically under 
computer control, then another degree of freedom is 
provided to enhance the accuracy of the tracking of 
camera 14, This can be done, for example, using one of 
the projectors manufactured by BARCO Projection Sys- 
tems of Kuurne. Belgium. Specifically, the patterns pro- 
jected onto screen 20 sut>sequent to the first pattern are 
transformed replicas of the first pattern. For exanple, a 
subsequent pattern may be a replica of the first pattern 
translated by a known amount, a replica of the first pat- 
tern dilated or contracted by a known amount, a replica 
of the first pattern subjected to a known affine transfor- 
mation, or a replica of the first pattern rotated by a 
known amount. Preferably, the patterns are projected 
sequentially in coordination with image capture by cam- ^ 
era 14. with each transformed replica projected onto j 
screen 20 during the entire capture by camera 1 4 of one * 
or more images of object 10 against the background of 
the pattern, so that the pattern is stable on the captured 
image. The extra degree of freedom provided by this 
dynamic alteration of the pattern compensates to a cer- 
tain extent for loss of resolution due to camera 14 being 
too close to screen 20 or too far from screen 20 to be 
tracked accurately relative to a static pattern of a given 
level of detail. For example, suppose the pattern rotates 
at a known uniform angular velocity. The changes in the 
positions of pattern features from one captured image to 
the next are functions, not only of the positional and ori- 
errtational coordinates of camera 14. txjt also of the 
radial distances of those features from the center of 
rotatk)n on screen 20. This provides information equiva- 
lent to the provision of a second pattern having a 
shorter scale length than the first pattern, so that a 
rotating pattern can be used to track camera 14 ait 
closer distances from screen 20 than a static pattern. 

If the transformation applied to the pattern is suffi- 
ciently simple, for example a rotation or a periodically 
oscillating translation, then the transformation may be 
effected by mechanical means, for example, by physi- 
cally rotating a "Gobo" mask inside a "Moving Light" 
projector. 

While the invention has been described with 
respect to a limited number of embodiments, it will be 
appreciated that many variations, modifications and 
other applications of the invention may be made. 

Where technical features mentioned in any daim 
are followed by reference signs, those reference signs 
have been included for the sole purpose of increasing 
the intelligibility of the claims and accordingly, such ref- 
erence signs do not have any limiting effect on the 
scope of each element identified by way of example by 
such reference signs. 

Claims 

1 . An improved video production method of the type in 
which a plurality of images of an object in a fore- 



EP0 876 055 A2 



8 




ground volume are capturec^Hcessively by a 

camera and superposed on a^Rual background, 
the position and orientation of the camera being 
determined using a first pattern featuring a first 
scale length and located at least partially beyond 
the object with respect to the camera, the improve- 
ment conrprising the steps of: 

(a) positioning a screen at least partially 
beyond the object with respect to the camera: 
and 

(b) projecting the first pattern onto said screen. 

The method of claim 1, wherein said projection is 
effected obliquely wrtii respect to said foreground 
volume. 
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tern is translated 



e to said first pattern. 



13. The method of claim 10, wherein said second pat- 
tern is dilated relative to said first pattern. 

14. The method of claim 10, wherein said second pat- 
tern is contracted relative to said first pattern. 

15. The method of claim 10, wherein said secorxj pat- 
tem is an affine transformation of said first pattern. 

16. The metiiod of claim 10, wherein laid second pat- 
tern is rotated relative to said first pattern. 



3. The method of claim 1. wherein said screen is 
transparent, and wherein said projection is effected 
from behind said screen with respect to said object. 20 

4. The method of claim 1 , wherein said screen has a 
surface featuring a chroma-key color, and wherein 
said projection is effected by directing light through 

a mask. 25 

5. The method of claim 4, wherein said light is sub- 
stantially white. 

6. The method of claim 1 , further conrprising the step 30 
of: 

(c) projecting a second pattern onto the screen. 



The method of daim 6, wherein said second pat- 
tern features a second scale length different from 
the first scale length. 
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The metfiod of daim 6, wherein said second pat- 
tern is displaced laterally with respect to the first 
pattern. 



40 



The method of claim 6. wherein tiie first pattern and 
said second pattern are projected on the screen 
together at least for the duration of the captijre of 45 
one of tiie images of the object 



10. The method of daim 6, wherein said second pat- 
tern is a transformed replica of the first pattern. 

1 1 . The method of claim 1 0, wherein the first pattern is 
projected during the capture of a first image of the 
object, and wherein said second pattern is pro- 
jected during the capture of a second image of the 
object immediately subsequent to the capture of the 
first image of the object. 

12. The method of daim 10. wherein said second pat- 
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(54) Method of providing badcground patterns for camera traddng 



(57) An improved method of tracking a camera (1 4) 
in dynamic chroma-keying of the type In which a fore- 
ground object is photographed against a multicolored 
chroma-key screen. The multicolor patterns are created 
by projection (32,30) on a monochrome screen (20), for 
exannple by projecting tight through a mask on a screen 
colored with the chroma-key color. The projection of 



multiple patterns can emulate the use of a single pattern 
having multiple levels of detail, and can enable lateral 
tracking of the camera. Sequentially projecting known 
transformations (for example, rotations) of a pattern 
adds an extra degree of freedom which further 
enhances the accuracy of the method. 
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